An evaluation of classification models for question topic categorization

We study the problem of question topic classification using a very large real-world Community Question Answering (CQA) dataset from Yahoo! Answers. The dataset contains 3.9M questions and these questions are organized in more than one thousand categories in a hierarchy. To our best knowledge, this is the first systematic evaluation of the performance of different classification methods on question topic classification as well as short texts. Specifically, we empirically evaluate the followings in classifying questions into CQA categories: 1) the usefulness of n-Gram features and bag-of-word features; 2) the performance of three standard classification algorithms (Naı̈ve Bayes, Maximum Entropy, and Support Vector Machines); 3) the performance of the state-of-the-art hierarchical classification algorithms; 4) the effect of training data size on performance; and 5) the effectiveness of the different components of CQA data, including subject, content, asker, and the best answer. The experimental results show what aspects are important for question topic classification in terms of both effectiveness and efficiency. We believe that the experimental findings from this study will be useful in real-world classification problems.